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Abstract 

Background: In 2013, a cohort of public health students participated in a 'flipped' Environmental and Occupational 
Health course. Content for the course was delivered through NextGenU.org and active learning activities were 
carried out during in-class time. This paper reports on the design, implementation, and evaluation of this novel 
approach. 

Methods: Using mixed-methods, we examined learning experiences and perceptions of the flipped classroom model 
and assessed changes in students' self-perceived knowledge after participation in the course. We used pre- and 
post-course surveys to measure changes in self-perceived knowledge. The post-course survey also included items 
regarding learning experiences and perceptions of the flipped classroom model. We also compared standard 
course review and examination scores for the 2013 NextGenU/Flipped Classroom students to previous years when 
the course was taught with a lecture-based model. We conducted a focus group session to gain more in-depth 
understanding of student learning experiences and perceptions. 

Results: Students reported an increase in knowledge and survey and focus group data revealed positive learning 
experiences and perceptions of the flipped classroom model. Mean examination scores for the 2013 NextGenU/ 
Flipped classroom students were 88.8% compared to 86.4% for traditional students (201 1). On a scale of 1-5 (1 = lowest 
rank, 5 = highest rank), the mean overall rating for the 2013 NextGenU/Flipped classroom students was 4.7/5 compared 
to prior years' overall ratings of 3.7 (2012), 4.3 (201 1), 4.1 (2010), and 3.9 (2009). Two key themes emerged from the focus 
group data: 1) factors influencing positive learning experience (e.g., interactions with students and instructor); and 2) 
changes in attitudes towards environmental and occupation health (e.g., deepened interest in the field). 

Conclusion: Our results show that integration of the flipped classroom model with online NextGenU courses can be an 
effective innovation in public health higher education: students achieved similar examination scores, but NextGenU/ 
Flipped classroom students rated their course experience more highly and reported positive learning experiences and 
an increase in self-perceived knowledge. These results are promising and suggest that this approach warrants further 
consideration and research. 
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^BMC 

Medical Education 



* Correspondence: lpg@sfu.ca 

'Faculty of Health Sciences, Simon Fraser University, Burnaby, BC V5A 1S6, 
Canada 

Full list of author information is available at the end of the article 

O© 201 4 Galway et al.; licensee BioMed Central ftd. This is an Open Access article distributed under the terms of the Creative 
BlOlVlGCl CGntrBl Commons Attribution License (http://creativecommons.Org/licenses/by/4.0), which permits unrestricted use, distribution, and 
reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain 
Dedication waiver (http://creativecommons.Org/publicdomain/zero/1.0/) applies to the data made available in this article, 
unless otherwise stated. 



Galway et al. BMC Medical Education 2014, 14:181 
http://www.biomedcentral.com/1472-6920/14/181 



Page 2 of 9 



Background 

The rapid increase in Internet access and advances in 
online technology over the last decade present an oppor- 
tunity to rethink the way we teach and learn in the 
context of public health higher education. The flipped 
classroom instructional model (also known as the 
inverted classroom) has emerged as a promising alterna- 
tive to conventional lecture-based teaching as it offers 
a framework for integrating emerging online learning 
technologies with active and collaborative learning. The 
flipped classroom model is a type of blended learning 
where in-class learning is integrated with online learning 
experiences [1,2]. A meta-analysis from the United States 
Department of Education in 2010 showed that blended 
learning, such as the flipped classroom, is more effective 
than either face-to-face (i.e. lecture-based instruction) or 
online learning alone [3]. This meta-analysis was focused 
on K-12 education. There has been limited research ex- 
ploring the flipped classroom model in the context of 
higher education (and none in the realm of public health 
higher education) highlighting an important knowledge 
gap. In an era of rising education costs and declining public 
funding for higher education, innovative approaches that 
foster positive learning experiences while taking advantage 
of emerging technologies and use both student and in- 
structor time more efficiently are called for [4]. 

The defining characteristic of the flipped classroom 
instructional model is that content and material are de- 
livered primarily outside of the classroom while in-class 
time is used "to work through problems, advance con- 
cepts, and engage in collaborative learning" [5]. Using 
online educational technologies to deliver content and 
material outside of the classroom frees up in-class time 
for active and collaborative application of content with 
the support of classmates and the instructor [2]. This 
model is designed to allow students to independently en- 
gage with materials on their own time and at their own 
pace, shifts focus from the instructor to the learner, and 
promotes active learning and problem-solving. 

The flipped classroom model involves more than shift- 
ing content delivery outside of class time [6]. It repre- 
sents a broader shift in how we think about the learning 
process. It is grounded in several interconnected theories 
of learning and pedagogy. The explicit attention to inter- 
active and collaborative learning draws on Piaget's theory 
of active learning which highlights that learning occurs 
when we act on and apply new ideas and concepts [7]. 
In terms of Bloom's influential (revised) taxonomy of 
thinking and learning, the flipped classroom enables both 
higher and lower levels of cognitive work [8,9]. More spe- 
cifically, students do lower level cognitive work, i.e., the 
acquisition of knowledge, independently and outside of 
class while higher-order cognitive work including know- 
ledge application, analysis, and synthesis occurs during 



class time with the support of peers and instructors. In 
our application of the flipped classroom design within the 
context of graduate level public health education, we also 
draw on Mesirow's theory of transformative adult learning 
[10] and Habermas' related theory of knowledge and hu- 
man interests [11]. We therefore integrate reflection, a key 
aspect of learning according Mesirow and an essential 
type of knowledge according to Habermas, into the course 
to complement the online content delivery and the in- 
class application of knowledge. 

There is no single or standard way to design and im- 
plement the flipped classroom instructional model in 
practice [6,12]. The means of delivering content and the 
ways in which face-to-face class time is used will vary 
with the characteristics of the students, background of 
the instructor, available resources and the subject matter. 
To date, electronic video-recordings (vodcasts) and pod- 
casts have been the primary means of content delivery, 
however online courses can also be used for this pur- 
pose. In our application of the flipped classroom model, 
we have used an environmental and occupational health 
course offered by NextGenU.org, a portal to free and 
accredited higher education [13]. Partnering with univer- 
sities, professional societies, and government organiza- 
tions, NextGenU creates online courses referred to as 
DOOHICHEs (Democratically-Open Outstanding Hybrids 
of Internet-aided, Computer-aided, and Human-aided 
Education, pronounced as "doohickey"). These DOO- 
HICHEs are competency-based and include high-quality 
learning resources, online peer activities and discussion 
forums, and quizzes. As of February 22 nd 2014, NextGenU 
has over 2,000 students registered in 105 countries, and 
has 130 trainings in development, primarily focusing on 
health sciences and public health education. The ways 
in which NextGenU builds on and differs from trad- 
itional education and Massive Open Online Courses can 
be reviewed at www.NextGenU.org. 

The goal of this study was to examine the impact of 
this NextGenU/Flipped classroom instructional model 
on self-perceived environmental and occupational health 
(EOH) knowledge and student learning experiences and 
perceptions of the instructional model. Our specific ob- 
jectives were: 1) to design and implement a master's level 
environmental and occupational health course (EOHC) in- 
tegrating a NextGenU DOOHICHE and the flipped class- 
room model, 2) to assess changes in students' self-reported 
EOH knowledge after participating in the course, and 3) to 
understand student learning experiences and perceptions 
of this NextGenU/Flipped classroom model. 

Methods 

Course design and implementation 

The first author designed the EOH DOOHICHE for 
NextGenU in 2012. The content and structure of the 
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DOOHICHE was based on core Master of Public Health 
competencies developed by the Association of Schools 
of Public Health (APSH) [14,15] and the Association 
of Schools of Public Health in the European Region 
(ASPHER) [16,17]. These core competencies "delineate 
fundamental knowledge, attitudes, and skills that every 
MPH student, regardless of their major field, should 
possess upon graduation" [18]. The final version was 
reviewed by an advisory committee of environmental 
health professionals, and endorsed by the International 
Society of Doctors for the Environment, Physicians for 
Social Responsibility, and Simon Fraser University's 
Faculty of Health Sciences. 

NextGenU's EOH DOOHICHE can be accessed for 
free in its entirety at www.NextGenU.org. Each of the 
course's nine modules (see Table 1), offered students the 
opportunity to learn content and material by engaging 



with a diversity of learning material (e.g., reports, journal 
articles, videos, websites) from reputable sources includ- 
ing universities, governments, professional societies, and 
peer-reviewed publications. The online course platform 
also included discussion forums, peer-to-peer activities, 
quizzes, and additional learning resources for those stu- 
dents wanting to explore certain subject areas more 
deeply. In-class sessions were held every other week, 
which gave students two weeks to complete the assigned 
modules. Students were assigned either one or two mod- 
ules for each two-week period. Within the two-week 
period, students could move through the materials at 
their own pace. After engaging with the learning mate- 
rials and completing peer activities in a given module, 
and before coming to the in-class sessions, students 
completed a quiz that helped students and the instructor 
identify poorly-understood areas. 



Table 1 Overview of the course 



Target participants MPH students 



Description of the course Students will gain familiarity with fundamental principles and general areas of knowledge that are important to the 

broad field of environmental health. Students will learn about approaches and tools used to recognize, assess, and 
manage environmental and occupational health hazards. This course also aims to expose students to numerous 
environmental and occupational health issues and to encourage critical thinking and reflection on these issues; we 
will consider what can be done about environmental and occupational health issues to ultimately protect and 
promote health and well-being. Finally, this course aims to inspire interest in the role of the environment in 
promoting and maintaining the health of populations in both local and global settings. 

Online learning modules from NextGenU DOOHICHE 

Module 1: Introduction to environmental and occupational health 

Module 2: Environmental and occupational hazards and their effects on human health and ecosystems 

Module 3: Principles of exposure assessment 

Module 4: Toxicology and epidemiology in environmental health 

Module 5: Risk assessment: Concepts and application 

Module 6: Risk management, communication, and regulation 

Module 7: Susceptibility, vulnerability, and inequality in environmental health 

Module 8: Environmental and occupational health case studies 

Module 9: Emerging perspectives in environmental health 

Learning activities used during class time: 

Module 1: Linking thinking for environmental health 

Module 2: Exposure assessment in action 

Module 3: Shipbreaking in Alang, India 

Module 4 and 5: Toxicology, epidemiology, and risk assessment problem set 

Module 6: Class debate- Should Canada apply the precautionary principle in environmental health policy and decision-making 

Module 7: A closer look at Air Quality Guidelines 

Module 8 and 9: Presentation and discussion of environmental health case studies 

Evaluation: 

Final grades were based on: 1) A final exam consisting of multiple choice and short answer application questions 

2) Four graded reflective responses written throughout the semester 

3) In-class and online participation 

4) A final group project exploring a selected environmental or occupational health issue 



^he online Learning modules in the NextGenU DOOHICHE can be accessed in full at http://www.nextgenu.org/course/view. php?id=52#0. 
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The general routine of in-class time included a mini- 
lesson addressing concepts or aspects that were identi- 
fied as challenging by students, along with a brief ques- 
tion and answer period to give students the opportunity 
to clarify any remaining aspects from the assigned mod- 
ules. The remaining in-class time (90 minutes) was used 
to carry out active learning activities. The learning activ- 
ities varied for each in-class session. Examples include: a 
toxicology problem set that students worked on in pairs; 
an occupational health case-study examined in small 
groups; and a whole-class debate on the topic of envir- 
onmental health decision-making (see Table 2 for a de- 
scription of three in-class activities). Over the course of 
the 13-week semester there were eight in-class sessions 
that ran for two hours. 

An additional requirement was writing four reflective 
responses. Students were instructed to reflect on the 
meaning and importance of the course material from 
their own perspective, to make links between the various 
aspects of EOH within the course, and to make links to 



the student's other public-health-related interests. The 
reflective responses acted as an ongoing conversation 
between the instructor and the students, and were also 
used by the instructor to identify areas of particular 
interest among the group and to influence and inform 
the direction of in-class activities. 

Finally, students completed a final exam at the end of 
the semester that included multiple choice and short 
answer questions. 

Study participants 

Study participants were MPH students (n = 11) enrolled 
in the 2013 spring session of an EOHC at a Canadian 
university. All participants were graduate students in 
their first or second year of study in the MPH program. 
An EOHC is offered in the second year of study and is a 
required core course of the MPH degree. Primary areas 
of interest among the students within the broad field of 
public health ranged from gender and health to social 
inequities and health. None of the students were focused 



Table 2 Description of a selection of in-class learning activities 

Example 1 



Activity name: 
Summary of activity: 



Example 2 
Activity name: 
Summary of activity: 



Example 3 
Activity name: 
Summary of activity: 



Linking thinking for environmental health 

Each student selects a health issue of interest. Working on their own, students create an influence diagram on 
a large blank piece of paper that explicitly illustrates the links between their selected health issue and the 
environment or environmental factors. In pairs, the students describe their influence diagram to one another. 
Finally, the instructor facilitates a discussion around the following questions "Was it difficult to integrate the 
environment and/or environmental factors into the influence diagram for your selected health issue? Why or 
why not?" 

Adapted from [1 9] 



Shipbreaking in Alang, India 

Before coming to class, students are instructed to watch the documentary 'Shopbreakers'. This documentary acted as 
background knowledge to complete the case-study described below. Students work on the case-study in groups 
of 3 or 4. 

You and your co-workers from Workplace Health Without Borders have recently returned from a visit to the 
shipbreaking port in Alang, India (the movie that you watched before coming to class this week served as your visit). 
Although it was a short trip, while in Alang you had the opportunity to observe the work setting and living conditions 
as well as to speak with many shipbreakers, port owners, the doctor, and other members of the community. Upon your 
return, your boss has asked for a review of what you saw and learned while visiting Alang. Specifically, she would like 
you and your colleagues to create a presentation that addresses the following questions: 

1. Who works in the shipbreaking yards? Describe relevant characteristics of workers in the shipbreaking yards. 

2. What are the major health hazards associated with shipbreaking identified during your visit? 

3. Use the following classification scheme (from the Canadian Centre for Occupational Hazards and Safety) to classify 
the hazards you have identified as: a) Biological; b) Chemical; c) Ergonomic (i.e. repetitive strain injury); d) Physical 
(noise, radiation); e) Psychosocial. 

4. For the chemical hazards identified, note the potential health implications using information provided by the Agency 
for Toxic Substances and Disease Registry at the following website: http://www.atsdr.cdc.gov/toxfaqs/index.asp. 

5. Drawing on your experience in Alang, what are possible research activities, interventions, or policy changes that 
our organization could implement to improve the current situation in Alang? Be sure to provide ample justification 
for your proposed research, intervention, or policy ideas. 



Toxicology, epidemiology, and risk assessment problem set 

Students work in pairs to solve problems applying toxicology, epidemiology, and risk assessment concepts. 
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specifically on environmental or occupational health nor 
enrolled in the EOH concentration of the MPH pro- 
gram. All of the students in the class agreed to partici- 
pate in the study. 

Evaluation and analysis 

We used a mixed-method approach including a pre- and 
post-survey, comparisons of overall course ratings and 
examination scores across years, and a post-course focus 
group session. A mixed-methods approach was used se- 
lected surveys are an effective tool for assessing pre and 
post student knowledge while focus groups are useful 
for gaining a more in-depth understanding of student 
perceptions and are also convenient and appropriate for 
non-sensitive topics. 

Pre- and post-course survey instruments were developed 
by the course instructor and reviewed by three colleagues. 
Participants electronically completed the pre-course survey 
one week prior to the beginning of the course and the 
post-course survey one week following course completion. 

To measure self-perceived knowledge, students were 
asked to rate their knowledge regarding EOH competen- 
cies. Students rated their knowledge relating to each 
competency (12 in total, see Figure 1) on a 5-point 
Likert scale (1 = strongly disagree; 5 = strongly agree). 
The self-perceived knowledge questions were identical 
in the pre- and post-course survey instruments. The 
post-course survey instrument also contained 10 items 
pertaining to student learning experiences and percep- 
tions of the flipped classroom, and one open-ended 
question where students were asked for general com- 
ments and/or feedback. 

To determine whether there was a statistically signifi- 
cant change in self-perceived EOH knowledge following 
participation in the class we used the Wilcoxon signed- 
rank test for paired data (5% significance level). This 
non-parametric test was selected because the data was 
non-normally distributed and ranked, and due to the 
small sample size. Descriptive statistics were used to 
report items pertaining to learning experiences and 
student perceptions of the flipped classroom model as 
measured in the post-course survey. For reporting of 
these items, agreement (strongly agree and agree) and 
disagreement (strongly disagree and disagree) were 
combined. 

We also compared overall course ratings and examin- 
ation scores for the NextGenU/Flipped classroom stu- 
dents against course ratings and examination scores for 
student who took a lecture-based EOHC in previous 
years. Specifically, we compared overall course ratings 
collected by the university from the 2013 students (n = 9 
who participated in the course and completed the stand- 
ard course review) with overall course ratings for stu- 
dents from 2012, 2011, 2010, and 2009 (n=130). To 



compare examination scores, we looked at scores from 
the course exam for the 2013 NextGenU/Flipped class- 
room students (n = 11) compared to scores for the 2011 
lecture-based EOHC students (n = 22). Examination re- 
sult data were only available for 2011. The exam con- 
tained 11 short answer questions, and was marked by 
the same individual. A Mann- Whitney U test (5% sig- 
nificance level) was used to assess whether there was a 
statistically significant difference in the examination 
scores for the 2013 vs. the 2011 students. 

Finally, to gain a more in-depth understanding of stu- 
dent perceptions of the NextGenU/Flipped classroom 
model, we (LPG) collected, audiotaped, and transcribed 
qualitative data from one 40-minute focus group session 
with all students in the course after the final in-class ses- 
sion. All focus group participants reviewed and signed a 
consent form. Open-ended queries addressed learning 
experiences and perceptions of the flipped classroom 
model in general. The data were analyzed using qualita- 
tive thematic analysis, selected because of the explora- 
tory and descriptive nature of this study. First, the 
transcribed text was read in its entirety to get a general 
overview and sense of the data. During a second reading 
of the transcript, the data were coded to identify and ex- 
plore themes. Major themes were therefore derived in- 
ductively from the data. Data were then summarized 
and representative excerpts from the focus group were 
identified to illustrate the major themes. 

All data collection was completed by the end of the 
2013 spring session of the university. Statistical analyses 
were conducted using R statistical software version 
2.13.0. Given the manageable length of the single focus 
group session, we used Microsoft Word to analyze the 
qualitative data [20]. 

Ethics approval was obtained from The University of 
British Columbia Ethics Committee. Recruitment was 
conducted by the lead author (LPG) during the first class 
session. All participants gave written informed consent 
and were assured of confidentiality and anonymity. Also, 
the voluntary nature of the study was underscored, and 
it was made clear to students that they could end their 
participation in the study at any point and that their 
marks in the class would not be affected by their deci- 
sion to participate in the study. 

Results 

Self-perceived knowledge assessment 

The response rate for the pre-and post-course survey 
was 100%. A comparison of self-perceived knowledge of 
EOH competencies before and after the course is pre- 
sented in Figure 1. A statistically significant increase was 
found in students' self-perceived knowledge for every 
competency (p-value < 0.05). 



Galway et al. BMC Medical Education 2014, 14:181 
http://www.biomedcentral.com/1472-6920/14/181 



Page 6 of 9 



I am able to describe Ihe direct and indirect human, ecological, and safety effects of Pie-course 
major environmental and occupational agents. Posi-course 




I can assess, evaluate, and propose solutions to environmental exposures and issues. Pre-course 

Post-course 




I can describe genetic, physiologic and psychosocial factors that affect susceptibility to Pre-course 
adverse health outcomes following exposure to environmental hazards. post-course 




I can describe regulatory programs and guidelines that control environmental health Pre-course 

issues. Post-course 




I can discuss various risk management and risk communication approaches and their Pre-course 
relation to issues of environmental justice and equity. Post-course 




I can explain the general mechanisms of toxicity in eliciting a toxic response to various Pre-course 
environmental exposures. Post-course 




I can specify current environmental risk assessment methods. Pre-course 

Posi-course 




I know and understand limitations of toxicology and of assessment measures for Pre-course 
environmental pollutions, and how they can affect assessment outcomes. Post-course 




I know and understand significant elements of the history of environmental and Pre-course 
occupational sciences. posi-course 




I know and understand the principles of exposure assessment to major environmental Pre-course 
contamination, current methods and tools. Post-course 




I know and understand the scale and the trends of current environmental health Pre-course 

problems Post-course 




I know and understand the specific design of epidemiologic studies in environmental Pre-course 

health. Post-course 


1 



1 1 1 1 

1 2 3 4 5 



Litert Scale (moan all participants) 

Figure 1 Comparison of pre- and post-course self-perceived knowledge of EOH competencies. 



Students' perceptions of the NextGenU/flipped 
classroom model 

Our results show that student perceptions of the course 
and the NextGenU/Flipped classroom instructional model 
were highly favorable overall. In response to the item "In 
the future, would you rather take a 'flipped' course than a 
traditional lecture-based course", 82% of students agreed 
or strongly agreed. All students agreed or strongly agreed 
with the statement, "The flipped classroom model was a 
different learning experience than other MPH courses." 
Also, the use of a NextGenU DOOHICHE for the online 
delivery of content and materials was favorably received. In 
response to the item, "I was comfortable with self-directed 
online learning through NextGenU", all students either 
agreed or strongly agreed (see Table 3 for a summary of 
results from the post-course survey). 

In response to the only open-ended question in the post- 
course survey "Do you have any additional feedback/ 
comments about the flipped classroom model?" stu- 
dents provided additional generally-positive feedback. 
One participant wrote: "I really enjoyed the structure 
and content of this course. I would say it was one of the 
best courses I've taken during the MPH! Also, it changed 



my perspective on online learning. I never thought on- 
line learning would be beneficial for me, but I will def- 
initely consider it if I decide to do continued studies 
later on." A second student wrote, "It was a great 
class and ignited an interest in enviro health that was not 
previously therel" 

Comparison of overall course ratings and examination scores 

The 2013 NextGenU/Flipped Classroom students rated 
their overall course experience more highly than those 
students who took the EOHC in previous years when 
a lecture-based model was used. On a scale of 1-5 
(1 = lowest rank; 5 = highest rank), the mean overall 
rating for the course in 2013 was 4.7/5 vs. prior years' 
overall ratings of 3.7 (2012), 4.3 (2011), 4.1 (2010), and 
3.9 (2009). We compared examination scores for the 
2013 NextGenU/Flipped classroom students to those for 
the 2011 lecture-based EOHC students. The mean test 
score was 88.8% for the 2013 students and 86.4% for the 
2011 students. A Mann-Whitney U test revealed no sta- 
tistically significant difference (p = 0.72) between examin- 
ation scores across the two groups . 
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Table 3 Descriptive statistics of post-course survey items focusing on learning experience and perceptions 

Survey item Agree/strongly Disagree/strongly Neutral (%) 

agree (%) disagree (%) 



was comfortable with self-directed learning through NextGenU. 100 0 0 

The online learning materials contributed to my learning. 100 0 0 

completed the activities and learning materials before in-class sessions. 100 0 0 

The quizzes encouraged completion of the online learning materials. 82 0 18 

n-class learning activities complemented online self-directed learning. 100 0 0 

The reflective responses contributed to my learning. 91 0 9 

nteraction with my instructor and other classmates contributed to my learning. 100 0 0 

The flipped classroom model (online learning plus in-person classroom interaction and 100 0 0 
problem-solving) was a different learning experience than other MPH courses. 

The flipped classroom model enabled more interaction with my instructor and classmates 82 0 18 
than did other MPH courses. 

In the future, I would rather take a flipped' course (blended online learning plus in-persons 82 9 9 
classroom interaction and problem-solving) than a traditional (lecture-based) course. 



NB: Students were asked to react to statements on a 5-point Likert scale where 7 = "Strongly disagree" and 5 = "Strongly agree." For reporting of these survey items, 
agreement (strongly agree and agree) and disagreement (strongly disagree and disagree) were combined. 



Key themes from focus group data 

Qualitative thematic analysis of the focus group data 
turned up two major themes: factors influencing positive 
learning experiences and changes in attitudes towards 
environmental and occupational health. These themes are 
discussed below along with excerpts from participants' 
responses. 

Students discussed multiple factors that contributed to 
their positive learning experiences. High amounts of 
interaction with other students and the instructor, small 
class size, the use of active in-class learning activities 
and reflective responses, and engagement with con- 
tent online before attending in-class sessions were all 
positively-cited aspects of the course and its design. 
Students commented that the class size and the flipped 
model contributed to a more interactive learning environ- 
ment and greater interaction with the instructor and their 
peers. This was recognized as deficient in previous 
courses: "...the discussions that we were able to have in 
class, by having a small class, that what we were able to 
get out of an in-class session was a lot more than a 3 hour 
lecture once per week of fifty people. " Several students also 
noted that they were unsure whether the flipped class- 
room model would be as effective with a larger class. For 
example, one student stated, "So if they are just going to 
try and flip i,t then just throw 50 people in there, I don't 
know if it would work." 

The focus group also highlighted the contribution of 
reflective responses to positive learning experiences. 
They allowed students to make connections between 
various aspects of the course with other areas of public 
health and student lives in general. One student stated: 
"And I found I would be like noticing environmental 
health news articles or stuff coming up, and you would 



think 'Oh! I could really try and incorporate this into my 
reflection'. A second student followed up on this com- 
ment stating: "Ya - that is what I found. I found the re- 
flections really encouraged me to be thinking about the 
course beyond the content, like beyond the actual read- 
ings because you don't really reflect about readings a lot 
of the time. But I was drawing on other things, and it 
kind of encouraged me to make those connections, those 
bigger connections. " 

Students also had positive comments about applying 
content through in-class learning activities. Students re- 
ported that the variety of learning activities used in the 
in-class sessions contributed to positive learning experi- 
ences. One student commented, "So I feel like I took in a 
lot of it, so that when I came to class I was cementing the 
knowledge that I just read. " Another student said, "every 
week there was something that was fresh and new and... 
interactive and I appreciated that. " 

Finally, several students noted that they were much 
more likely to complete readings and engage with con- 
tent before coming to class in the flipped classroom 
model than the traditional lecture-based model. One 
student stated, "it is really easy to justify that you don't 
have time to do readings, but for this set-up I think that 
is was just like, something I needed to get done. " 

When students began this course, there was very little 
interest in the field of environmental health and occupa- 
tional health in the group. For example, none of the stu- 
dents were part of the EOH concentration option 
available for the MPH students at the test-site university. 
The majority of students noted that they would not have 
taken the course had it not been a required core course 
for their degree. However, a major theme that emerged 
from the focus group was that this course contributed to 
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greater interest in environmental health and environ- 
mental health issues in general. One student noted about 
the course: "It gave me a huge appreciation for environ- 
mental health, it really did truly." A second student 
agreed and stated: "I feel like I got a tremendous appreci- 
ation for environmental health and I got a really good 
grounding in environmental and occupation health and 
so that is why, you know, I really really liked the course. " 

Discussion 

This paper reported on the design, implementation, 
and evaluation of a master's level EOHC that inte- 
grated two emerging instructional models: NextGenU's 
DOOHICHE and the flipped classroom. Our results 
suggest that this innovative approach fostered learning 
and provided positive learning experiences for this 
small group of graduate students. To quote one of the 
students, "This was a successful experiment!" Students 
also expressed a preference for the flipped classroom 
instructional model over the traditional lecture-based 
model, with most (82%) agreeing that, "In the future, I 
would rather take a 'flipped' course (blended online learn- 
ing plus in-persons classroom interaction and problem- 
solving) than a traditional (lecture-based) course. In 
addition, the instructor (LPG) had positive experiences 
regarding the design, implementation, and overall out- 
comes of the course. 

This instructional model should be considered for 
more widespread experimentation in the context of pub- 
lic health higher education and beyond. It is worth not- 
ing that the flipped classroom, blended learning, and 
NextGenUs DOOHICHEs are three of many emerging 
approaches that are student-centered, promote applica- 
tion and collaboration, and optimize face-to-face time 
and complementary educational technologies [6]. We 
urge instructors interested in moving away from conven- 
tional lecture-based teaching to consider the wide range 
of instructional models available and to select the model 
which bests suits the course content, student needs, and 
available resources to optimally facilitate learning. 

For those instructors considering applying the flipped 
classroom instructional model, we caution that 'flipping' 
the classroom is not simply about shifting lectures out- 
side of the classroom. Content delivery is "just one small 
piece of the overall learning experience..." [21]. 'Flipping' 
the classroom involves seeing students as active learners, 
shifting control of both learning and the classroom from 
the instructor to the students; it should promote a focus 
on higher-order cognitive work. Additionally, research 
shows that, "for blended learning environments to be 
successful, it is important to structure the face-to-face 
and the online portions of the learning experience so 
that they coherently support one another" [22]. Educators 
should think purposefully about course design, develop 



effective learning activities that engage students, encourage 
reflections, and complement online content, and take max- 
imal advantage of invaluable face-to-face class time. 

Further, we encourage instructors to think creatively 
about how to use emerging educational technologies 
in their teaching. For this work, we have employed a 
NextGenU DOOHICHE to deliver content online, but 
other opportunities for innovation in blended learning 
also exist. The use of open online courses within the 
flipped classroom, and blended learning more broadly 
can reduce the burden of course re-design on instructors 
and institutions by most-fully utilizing already-existing 
and free courses. 

Although we agree with others who suggest that that 
the flipped classroom model has the potential to influ- 
ence the landscape of higher education [23-25], we also 
note that more research is needed to evaluate the im- 
pacts of this model on teaching and learning experiences 
and to better understand the specific characteristics of 
'flipped' courses that lead to positive impacts. It is also 
important for instructors to share their experiences and 
course design to ensure that we are collectively capitaliz- 
ing on lessons learned in implementation. Finally, more 
research is needed to understand the role of reflection in 
the flipped classroom instructional model. Reflection 
which plays a key role in adult learning, has not been ex- 
plicitly considered within the flipped classroom model 
[10]. Students in the flipped classroom environments 
"need to have more space to reflect on their learning ac- 
tivities so that they can make necessary connections to 
course content" [1]. Qualitative evidence from this small 
cohort, as well as the opinions of the course instructor, 
suggest that reflection is an important yet overlooked 
element of applying the flipped classroom model. 

There are limitations that should be noted. We cannot 
exclude the possibility that students answered questions 
in a socially desirable way nor that the positive percep- 
tions of this instructional model were attributable to it 
being a different experience than other courses offered 
to the students. A major limitation of this study is the 
small sample size. As this was a pilot study, the sample 
size was simply the number of students that participated 
in the class. Finally, comparisons of overall course rat- 
ings and examination scores should be interpreted with 
caution. Although the courses were designed around the 
same MPH core competencies, there were important dif- 
ferences across years that we have not controlled for in 
the design or analysis. Specifically, different instructors 
taught the course, class sizes were higher in previous 
years (as high as 50) compared to 11 in the 2013, and re- 
flection was only included as part of the curriculum in 
the flipped classroom EOHC. These factors could cer- 
tainly influence both overall course ratings and examin- 
ation scores. 
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Conclusion 

The lecture-based teaching model continues to domin- 
ate higher education despite massive advances in online 
access and technology and developments in pedagogical 
theory [26,27]. Our results show that the flipped class- 
room model can have positive impacts on learning and 
learning experiences in public health higher education. 
Our data suggest that the use of a DOOHICHE from 
NextGenU was an effective and efficient means of con- 
tent delivery and could facilitate more widespread appli- 
cation of the flipped classroom in public health higher 
education. 
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